Skip to content

feat: support DSV4 device prefix cache (SWA + C4 + C128).#2008

Open
Kang-Meng wants to merge 1 commit into
xLLM-AI:mainfrom
Kang-Meng:feat_dsv4_prefixcache
Open

feat: support DSV4 device prefix cache (SWA + C4 + C128).#2008
Kang-Meng wants to merge 1 commit into
xLLM-AI:mainfrom
Kang-Meng:feat_dsv4_prefixcache

Conversation

@Kang-Meng

@Kang-Meng Kang-Meng commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

feat: support DSV4 device prefix cache (SWA + C4 + C128)

Commit: d54b1af · Branch: feat_dsv4_prefixcache · Repo: xllm
Diff: 30 files, +1242 / −576

Background

DeepSeek-V4-Flash (DSV4) previously could not use the device prefix cache.
master.cpp carried a hard guard that, on detecting a deepseek_v4 model type,
force-set enable_prefix_cache=false and logged a warning. This PR removes that
guard and completes CompositeBlockManager + prefix-cache support for DSV4's
three-segment KV layout (SWA + C4 + C128), so DSV4 can actually hit the
device-side prefix cache.

What changed

  1. Removed the DSV4 fallback guard (master.cpp, −11 lines).
    DSV4 now takes the normal composite prefix-cache path instead of being
    silently disabled.

  2. LeafCombination classification in CompositeBlockManager
    (largest change; composite_block_manager.cpp +630). The leaf shape is
    classified once at construction and cached; the sequence-level orchestrators
    dispatch on it to pick the trim/mount strategy:

    • FLAT_KV — plain KV (normal / Qwen), no trim
    • FLAT_KV_LINEAR — Qwen3.5-Next GDN (KV + LINEAR restore)
    • SWA_COMPRESSEDDSV4 (SWA + C4 + C128): cross-leaf min,
      C128-stride clamp, SWA tail-continuity, exact-repeat pop
    • UNSUPPORTED — prefix cache off / xtensor

    Adds cache_full_blocks_for_sequence to incrementally insert each leaf's
    newly-forwarded full blocks into its own prefix cache.

  3. Per-stride block-hash chains (sequence.h/.cpp).
    DSV4 admission probes SWA/C4/C128 back-to-back each tick with different
    strides (base / 4×base / 128×base). Each stride now keeps its own
    incrementally-extended hash chain, so switching strides no longer discards
    and rebuilds the whole chain. Backed by std::map for pointer stability, so
    a Slice from block_hashes() survives inserts of other strides.

  4. LinearStatePrefixCache made gap-tolerant
    (linear_state_prefix_cache.cpp +229). SWA slides out middle window blocks
    and LINEAR deeper hits subsume earlier ones, so match()/insert() tolerate
    middle misses (placeholders): the compute path walks the chain from token 0
    rather than seeding from the previous block, placeholders advance the cursor
    without being stamped/emplaced, and trailing misses are trimmed. The base
    PrefixCache remains solid-prefix.

  5. Interface cleanup: dropped the size_t* matched_tokens out-param from
    match() (reach now expressed as returned.size() * block_size_); added an
    LRU-neutral contains().

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@Kang-Meng
Kang-Meng force-pushed the feat_dsv4_prefixcache branch 2 times, most recently from 44a19c9 to ab71000 Compare July 23, 2026 05:52
@Kang-Meng
Kang-Meng marked this pull request as ready for review July 23, 2026 05:54
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.


Node* get_first() { return lst_front.next; }

// pop out node to the back of the LRU list

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为什么移除了,原来的许多注释

yingxudeng
yingxudeng previously approved these changes Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants